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Program description 


Saxon Elementary School Math, published by Harcourt Achieve, 
is a core curriculum for students in kindergarten through 
grade 5. A distinguishing feature of Saxon Elementary School 
Math is its use of a distributed approach, as opposed to a 
chapter-based approach, for instruction and assessment. The 
program is built on the premise that students learn best when 
instruction is incremental and explicit, previously learned con- 
cepts are continually reviewed, and assessment is frequent and 


cumulative. At each grade level, math concepts are introduced, 
reviewed, and practiced over time in order to move students 
from understanding to mastery to fluency. For grades K-3, the 
Saxon Elementary School Math curriculum emphasizes hands- 
on activities and teacher-directed math conversations that 
engage students in learning. The curriculum for grades 4-5 also 
uses math conversations to introduce new concepts, but shifts 
the focus to student-directed learning. 


Research 


One study of the Saxon Elementary School Math program met 
the What Works Clearinghouse (WWC) evidence standards 
with reservations. The study included students in grades 1-8 


from a range of socioeconomic backgrounds and attending 342 
schools across the state of Georgia. This report focuses only on 
findings for grades 1-5. 1 



Effectiveness Saxon Elementary School Math was found to have no discernible effects on mathematics achievement. 



Mathematics achievement 
Rating of effectiveness No discernible effects 
Improvement index 2 Average: +7 percentile points 

Range: +4 to +9 percentile points 



1. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

2. These numbers show the average and range of improvement indices across all findings in the study. 
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Developer and contact 

Saxon Elementary School Math was developed by Saxon Pub- 
lishers, an imprint of Harcourt Achieve. Address: 6277 Sea Har- 
bor Drive, Orlando, Florida 32887. Email: info@SaxonPublishers. 
com. Web: http://saxonpublishers.harcourtachieve.com . 
Telephone: (800) 284-7019. 

Scope of use 

The first Saxon textbook, Saxon Algebra, was published in 
1979 by John Saxon for junior college students. In 1980, a high 
school version, Algebra 1, was published. In 1981, the program 
was tested by 20 teachers with approximately 1,400 students. 

By 1993, the company had become Saxon Publishers and 
developed programs for kindergarten through high school. 
Information is not available on the number or demographics of 
students, schools, or districts using this intervention. 

Teaching 

Daily lessons in grades 1-3 consist of three components: 1) the 
meeting, 2) the math lesson, and 3) the written practice, which 
includes guided class practice and homework. Atypical lesson 
begins with the meeting where students engage in various prac- 
tical activities (for example, understanding calendars), and enter 
into math conversations and dialogue with the class and teacher 
to communicate their understanding of math concepts. Follow- 
ing the meeting, the teacher begins the math lesson in which 
new concepts are introduced. Hands-on activities are incorpo- 
rated into the math lesson to encourage student involvement 



and further the learning of new concepts. The math lesson is 
followed by written practice, which includes teacher-facilitated 
guided class practice of newly and previously learned concepts. 
The day’s homework is completed by the students indepen- 
dently. Cumulative and written assessments occur every five les- 
sons. In kindergarten, these components may be separated into 
different sessions, and assessments are conducted as individual 
interviews between the teacher and individual students. 

For grades 4 and 5, a daily lesson consists of four compo- 
nents: 1) warm-up, 2) lesson introducing new math concept, 3) 
practice on new concept, and 4) mixed practice including new 
and previously learned concepts. Students are introduced to 
concepts incrementally, given opportunities for continual review 
and practice, and assessed cumulatively and frequently. An 
assessment score of 80% or lower indicates a need for remedia- 
tion, and provision for remediation is part of the program. 

Cost 

Saxon Elementary School Math grades 1-3 can be ordered as a 
24-student or 32-student kit that includes all of the teacher, les- 
son, classroom and student materials. The student kits range from 
over $600 to over $800 depending on the size of the kit. Individual 
kit components, such as manipulatives, workbooks, student texts, 
teacher manuals, and materials in Spanish, can be purchased 
separately. Grades 4 and 5 have a separate student edition 
($50-$55) and teacher manual set ($185). Other ancillary materi- 
als, such as black line master books, fact practice workbooks, 
and a test practice generator, can be purchased separately. 



Seven studies reviewed by the WWC investigated the effects of 
Saxon Elementary School Math. One study (Resendez & Manley, 
2005) was a quasi-experimental design that met WWC evidence 
standards with reservations. The remaining six studies did not 
meet WWC evidence screens. 

Resendez & Manley (2005) conducted a retrospective study 
that included 170 intervention schools in Georgia and 172 
comparison schools that were matched to the intervention 



schools on student demographics, geographical location, and 
baseline math performance on Georgia’s Criterion-Referenced 
Competency Test (CRCT). The intervention schools used the 
Saxon Elementary School Math program recommended for each 
grade level in grades 1-8 between 2000 (or prior) and 2005. The 
comparison schools used a variety of non-Saxon curricula. The 
majority of comparison schools used traditional basal math cur- 
ricula. One third of the schools used a mix of basal, investigative, 
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and other approaches, and a small percentage used an investiga- findings for grades 1-5 on Georgia’s Criterion-Referenced 
tive approach to teaching math. This intervention report presents Competency Test (CRCT). 



Findings 

The WWC review of elementary school mathematics curriculum- 
based interventions addresses student outcomes in overall 
mathematics achievement. 

Resendez & Manley (2005) reported no significant effects 
of the Saxon Elementary School Math program on overall 
mathematics achievement at grades 1-5. Using school-level 
data provided by the authors, the WWC confirmed that Saxon 
Elementary School Math did not have a statistically significant 
or substantively important effect on math achievement at each 
grade level from first to fifth grade. Based on this study finding, 
the WWC categorized Saxon Elementary School Math as having 
no discernible effects on overall math achievement. 

The WWC also calculated effect sizes and significance 
levels for subtests of the Georgia CRCT using school-level data 
provided by the authors. The WWC found significant effects of 



Saxon Elementary School Math on patterns, relations, and Alge- 
bra at Grade 2. For grade 4 the WWC found significant effects of 
Saxon Elementary School Math on computations and estimation, 
and problem solving. Subtest results were not included in the 
WWC’s rating of effectiveness. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as: positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors; the quality of the research 
design, the statistical significance of the findings (as calculated 
by the WWC 3 ), the size of the difference between participants 
in the intervention and the comparison conditions, and the con- 
sistency in findings across studies (see the WWC Intervention 
Rating Scheme) . 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study as 
well as an average improvement index across studies (see the 
WWC Improvement Index Technical Paper) . The improvement 
index represents the difference between the percentile rank 
of the average student in the intervention condition versus 
the percentile rank of the average student in the comparison 
condition. Unlike the rating of effectiveness, the improvement 
index is based entirely on the size of the effect, regardless of 
the statistical significance of the effect, the study design, or the 
analysis. The improvement index can take on values between 



-50 and +50, with positive numbers denoting favorable results. 
The average improvement index for math achievement is 
+7 percentile points for the one study of Saxon Elementary 
School Math, with a range of +4 to +9 percentile points across 
findings. 

Summary 

The WWC reviewed seven studies on Saxon Elementary School 
Math. One of these studies met WWC standards with reserva- 
tions and the remaining studies did not meet WWC evidence 
screens. This study found no discernible effects on mathematics 
achievement. The evidence presented in this report is limited 
and may change as new research emerges. 



3. The level of statistical significance was reported by the study authors, or where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple 
comparisons. For an explanation, see the WWC Tutorial on Mismatch. See the WWC Intervention Rating Scheme for the formulas the WWC used to calculate the statistical significance. In 
the case of the Saxon Elementary School Math report, no corrections for clustering or multiple comparisons were needed. 
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For more information about specific studies and WWC calculations, please see the WWC Saxon Elementary 
School Math Technical Appendices . 



4. Does not use a strong causal design: the study, which used a quasi-experimental design, did not establish that the comparison group was equivalent to 
the treatment group at the baseline in a pretest measure of math achievement. 

5. Does not use a strong causal design: the study did not use a comparison group. 

6. Does not use a strong causal design: this is a qualitative study. 

7. Disruption: the study, which used a quasi-experimental design, demonstrated problems with disruption or contamination. 



WWC Intervention Report Saxon Elementary School Math 



October 26, 2006 



Appendix 



Appendix A1 


Study characteristics: Resendez & Manley, 2005 (quasi-experimental design) 


Characteristic 


Description 


Study citation 


Resendez, M., & Manley, M. A. (2005). The relationship between using Saxon Elementary and Middle School Math and student performance on Georgia Statewide Assess- 
ments. Orlando, Fla.: Harcourt Achieve. 


Participants 


The participants in this study were students in grades 1-8 in 170 intervention schools and 172 comparison schools that were matched on student demographics, geographical 
location, and baseline math performance on Georgia's CRCT. This intervention report focuses only on findings for grades 1-5, because grades 6-8 are outside of the scope 
of this review. 1 The authors selected Georgia schools that used the Saxon Elementary School Math curriculum between 2000 and 2005. The sample was obtained from the 
Georgia Department of Education (GDE). The authors note that per state policy, only school-level data could be released. Data for the intervention group came from 85 schools 
for first grade, 85 schools for second grade, 83 schools for third grade, 79 schools for fourth grade and 79 schools for fifth grade. Data for the comparison group came from 
144 schools for first grade, 144 schools for second grade, 135 schools for third grade, 131 schools for fourth grade, and 129 schools for fifth grade. The number of schools per 
grade is not mutually exclusive. Some of the schools contained multiple grades so the numbers presented do not represent distinct clusters of schools. 


Setting 


The sample schools were distributed across the state of Georgia and represented a mixture of rural, urban, and suburban communities. The gender and racial compositions of 
the schools were similar in the intervention schools and comparison schools, with roughly equal gender distribution and more than half of the students white. More than half of 
the students were educationally disadvantaged, 16% had disabilities, 3% had limited English proficiency, and 6% were gifted in both study conditions. 


Intervention 


The Saxon Elementary School Math curriculum was used as a core curriculum in the intervention schools. The elementary schools in the sample used the version of the Saxon 
Elementary School Math program that was appropriate for each grade level, and participating schools had used the program for an average of three years (range 1-15 years). 


Comparison 


The schools in the comparison group used a mixture of non -Saxon curricula. More than half of the schools in the comparison group used basal math curricula with chapter- 
based approaches to teaching math. Five percent of the schools used curricula with an investigative approach. The remaining third of the schools used curricula that were a 
mix of basal, investigative, computer-based approaches. The authors report no significant differences in baseline math performance between the Saxon and non -Saxon schools. 


Primary outcomes 
and measurement 


The outcome measure was the Georgia’s Criterion-Referenced Competency Test (CRCT) which assesses competency in number sense and numeration, geometry and 
measurement, patterns and relations/algebra, statistics and probability, computation and estimation, and problem solving. Fourth-grade students were tested in each school 
year from 1999-2000 to 2004-05. First-grade, second-grade, third-grade, and fifth-grade students were tested in the spring of school years 2001-02, 2003-04, and 
2004-05. All posttest scores are from spring 2005 (see Appendix A2 for more detailed descriptions for outcome measures). 


Teacher training 


No information was provided regarding the teacher training for the intervention. 



1. Results from grades 6-8 are being reviewed as part of the WWC Middle School Mathematics review. 
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Appendix A2 Outcome measures in the mathematics achievement domain 



Outcome measure 


Description 


Georgia’s Criterion- 
Referenced Competency 
Test (CRCT) 1 


As cited in Resendez & Manley (2005), the CRCT is a criterion-referenced test which is referenced to Georgia’s Quality Core Curriculum Goals. According to the Georgia 
Department of Education, the CRCT is a multiple-choice test that is valid and reliable for Georgia's public school students. 2 The CRCT math scores range from 150-450, with 
scores below 300 not meeting standards and scores above 350 exceeding standards. The criteria for meeting the standards vary by objective and grade level. Five objectives 
are covered by the test: 1) number and number sense, 2) geometry and measurement, 3) patterns, relationships, and Algebra, 4) computation and estimation, and 5) problem 
solving. The cut points are set by the state and take into account the difficulty of the specific objective. 



1. The original CRCT scores shown in the report are by objective. Upon request from the WWC, the author calculated the mean overall score across all objectives, controlling for pretest, for each 
grade. 

2. Georgia Department of Education, (n.d.). Criterion-Referenced Competency Tests. Retrieved on September 13, 2006 from http://www.doe.k12.aa.us/curriculum/testing/crct.asp . 
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Appendix A3 Summary of study findings included in the rating for mathematics achievement 1 



Outcome measure 


Study 

sample 




Authors’ findings from the study 

Mean outcome 
(standard deviation 2 ) 




WWC calculations 




Sample size 
(schools) 


Saxon 

group 3 


Comparison 

group 3 


Mean difference 4 
(Saxon - 
comparison) 


Effect size 5 


Statistical 
significance 6 
(at a = 0.05) 


Improvement 

index 7 




GCRCT: Overall % students 
meeting objectives 


Grade 1 


Resendez & Manley 2005 (quasi-experimental design) 8 

229 86.26 85.2 1.06 

(6.60) (6.80) 


0.16 


ns 


+6 


Average 9 for math achievement (Resendez & Manley, 2005): Grade 1 








0.16 


ns 


+6 


GCRCT: Overall % students 


Grade 2 


229 


88.31 


86.86 


1.45 


0.21 


ns 


+8 


meeting objectives 






(6.39) 


(7.35) 










Average 9 for math achievement (Resendez & Manley, 2005): Grade 2 








0.21 


ns 


+8 


GCRCT: Overall % students 


Grade 3 


218 


86.94 


85.93 


1.01 


0.15 


ns 


+6 


meeting objectives 






(6.50) 


(7.15) 










Average 9 for math achievement (Resendez & Manley, 2005): Grade 3 








0.15 


ns 


+6 


GCRCT: Overall % students 


Grade 4 


210 


73.92 


71.39 


2.53 


0.24 


ns 


+9 


meeting objectives 






(8.51) 


(11.83) 










Average 9 for math achievement (Resendez & Manley, 2005): Grade 4 








0.24 


ns 


+9 


GCRCT: Overall % students 


Grade 5 


208 


82.46 


81.66 


0.80 


0.10 


ns 


+4 


meeting objectives 






(6.94) 


(8.93) 










Average 9 for math achievement (Resendez & Manley, 2005): Grade 5 








0.10 


ns 


+4 


Domain average 9 for math achievement across all grades 








0.17 


na 


+7 



ns = not statistically significant 
na = not applicable 



1 . This appendix reports findings considered for the effectiveness rating and the average improvement indices. Subtest findings from the same studies are not included in these ratings, but are reported in Appendix A4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

3. The intervention group and control group means are pretest-adjusted means provided by the authors and differ from what is in the original study. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. The effect sizes were computed based on school-level data, which were likely to be larger than effect sizes based on student-level data. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the percentile rank of the average student in the comparison condition. The improvement index can 
take on values between -50 and +50, with positive numbers denoting favorable results. 

8. The level of statistical significance was reported by the study authors, or where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the cluster- 
ing correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Resendez & Manley (2005), no corrections 
for clustering or multiple comparisons were needed. 

9. This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated 
from the average effect size. 
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Appendix A4 Summary of subtest findings for mathematics achievement 1 









Authors’ findings from the study 

Mean outcome 
(standard deviation 2 ) 




WWC calculations 














Mean difference 4 




Statistical 






Study 


Sample size 


Saxon 


Comparison 


(Saxon- 




significance 6 


Improvement 


Outcome measure 


sample 


(schools) 


group 3 


group 3 


comparison) 


Effect size 5 


(at a = 0.05) 


index 7 



Resendez and Manley 2005 (quasi-experimental design) 8 



GCRCT: Number and 
number sense 


Grade 1 


229 


89.53 

(6.31) 


88.52 

(7.00) 


1.01 


0.15 


ns 


+6 


GCRCT: Geometry and 
measurement 


Grade 1 


229 


90.34 

(5.83) 


90.29 

(5.70) 


0.05 


0.01 


ns 


0 


GCRCT: Patterns, 
relations, and Algebra 


Grade 1 


229 


87.88 

(6.99) 


86.28 

(6.61) 


1.60 


0.24 


ns 


+9 


GCRCT: Computation 
and estimation 


Grade 1 


229 


78.93 

(9.54) 


77.43 

(10.10) 


1.50 


0.15 


ns 


+6 


GCRCT: Problem solving 


Grade 1 


229 


84.64 

(7.30) 


83.49 

(8.39) 


1.15 


0.14 


ns 


+5 


GCRCT: Number and 
number sense 


Grade 2 


229 


88.57 

(6.80) 


86.62 

(8.38) 


1.95 


0.25 


ns 


+10 


GCRCT: Geometry and 
measurement 


Grade 2 


229 


91.46 

(6.18) 


92.36 

(5.41) 


-0.90 


-0.16 


ns 


-6 


GCRCT: Patterns, 
relations, and Algebra 


Grade 2 


229 


87.05 

(7.43) 


83.58 

(9.63) 


3.47 


0.39 


Statistically 

significant 


+15 


GCRCT: Computation 
and estimation 


Grade 2 


229 


86.93 

(7.13) 


85.83 

(7.82) 


1.10 


0.15 


ns 


+6 


GCRCT: Problem solving 


Grade 2 


229 


87.54 

(7.48) 


85.93 

(8.28) 


1.61 


0.20 


ns 


+8 


GCRCT: Number and 
number sense 


Grade 3 


218 


89.74 

(6.29) 


88.24 

(7.02) 


1.50 


0.22 


ns 


+9 


GCRCT: Geometry and 
measurement 


Grade 3 


218 


93.6 

(4.50) 


92.24 

(6.22) 


1.36 


0.24 


ns 


+10 


GCRCT: Patterns, 
relations, and Algebra 


Grade 3 


218 


86.26 

(6.67) 


85.9 

(7.12) 


0.36 


0.05 


ns 


+2 



(continued) 



WWC Intervention Report Saxon Elementary School Math 



October 26, 2006 



8 





Appendix A4 Summary of subtest findings for mathematics achievement 1 (continued) 



Authors’ findings from the study 



Mean outcome 

(standard deviation 2 ) WWC calculations 



Outcome measure 


Study 

sample 


Sample size 
(schools) 


Saxon 

group 3 


Comparison 

group 3 


Mean difference 4 
(Saxon - 
comparison) 


Effect size 5 


Statistical 
significance 6 
(at a = 0.05) 


Improvement 

index 7 




GCRCT: Statistics and probability 


Grade 3 


218 


87.13 

(7.21) 


85.83 

(7.98) 


1.30 


0.17 


ns 


+7 


GCRCT: Computation 
and estimation 


Grade 3 


218 


86.81 

(7.80) 


85.71 

(8.02) 


1.10 


0.14 


ns 


+5 


GCRCT: Problem solving 


Grade 3 


218 


78.11 

(10.12) 


77.64 

(10.69) 


0.47 


0.04 


ns 


+2 


GCRCT: Number and 
number sense 


Grade 4 


210 


71.47 

(10.32) 


70.85 

(14.39) 


0.62 


0.05 


ns 


+2 


GCRCT: Geometry and 
measurement 


Grade 4 


210 


79.22 

(8.93) 


78.16 

(11.13) 


1.06 


0.10 


ns 


+4 


GCRCT: Patterns, 
relations, and Algebra 


Grade 4 


210 


69.76 

(8.77) 


67.7 

(11.07) 


2.06 


0.20 


ns 


+8 


GCRCT: Statistics and probability 


Grade 4 


210 


82.15 

(9.05) 


80.17 

(10.82) 


1.98 


0.19 


ns 


+8 


GCRCT: Computation 
and estimation 


Grade 4 


210 


73.12 

(10.30) 


67.65 

(14.75) 


5.47 


0.41 


Statistically 

significant 


+16 


GCRCT: Problem solving 


Grade 4 


210 


67.81 

(9.87) 


63.83 

(14.44) 


3.98 


0.31 


Statistically 

significant 


+12 


GCRCT: Number and 
number sense 


Grade 5 


208 


79.74 

(9.55) 


77.31 

(11.51) 


2.43 


0.22 


ns 


+9 


GCRCT: Geometry and 
measurement 


Grade 5 


208 


80.77 

(9.01) 


81.54 

(9.88) 


-0.77 


-0.08 


ns 


-3 


GCRCT: Patterns, 
relations, and Algebra 


Grade 5 


208 


76.16 

(9.37) 


74.56 

(12.11) 


1.60 


0.14 


ns 


+6 


GCRCT: Statistics and probability 


Grade 5 


208 


79.82 

(7.71) 


81.52 

(9.65) 


-1.70 


-0.19 


ns 


-8 


GCRCT: Computation 
and estimation 


Grade 5 


208 


88.74 

(6.56) 


86.62 

(8.55) 


2.12 


0.27 


ns 


+11 



(continued) 
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Appendix A4 Summary of subtest findings for mathematics achievement 1 (continued) 









Authors’ findings from the study 

Mean outcome 
(standard deviation 2 ) 




WWC calculations 














Mean difference 4 




Statistical 






Study 


Sample size 


Saxon 


Comparison 


(Saxon- 




significance 6 


Improvement 


Outcome measure 


sample 


(schools) 


group 3 


group 3 


comparison) 


Effect size 5 


(at a = 0.05) 


index 7 



GCRCT: Problem solving Grade 5 208 89.55 88.43 1.12 0.16 ns +6 

(6i35) (743) 

ns = not statistically significant 

1 . This appendix presents subscale findings for measures that fall in math achievement. Total scale scores were used for rating purposes and are presented in Appendix A3. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

3. The intervention group and control group means are pretest-adjusted means provided by the authors and differ from what is in the original study. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. The effect sizes were computed based on school-level data, which were likely to be larger than effect sizes based on student-level data. The effect sizes may differ from those presented by the authors because the authors calculated 
Cohen’s c/ using the proportion of variance, while the WWC calculated a Hedges' g using the adjusted means, unadjusted standard deviations, and sample sizes for each group using additional data provided by the authors. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The WWC used additional data provided by the authors to calculate significance. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the average student in the comparison condition. The improvement index can take on values between 
-50 and +50, with positive numbers denoting favorable results. 

8. The level of statistical significance was reported by the study authors, or where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings 
not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate 
statistical significance. In the case of Resendez & Manley (2005), no corrections for clustering or multiple comparisons were needed. 
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Appendix A5 Saxon Elementary School Math rating for the mathematics achievement domain 



The WWC rates the effects of an intervention in a given outcome domain as: positive, potentially positive, mixed, no discernible effects, potentially negative, or negative. 1 

For the outcome domain of math achievement, the WWC rated the Saxon Elementary School Math program as having no discernible effects. It did not meet the 
criteria for other ratings (positive effects, potentially positive effects, mixed effects, potentially negative effects, or negative effects) because the single study that met 
WWC evidence standards with reservations did not show statistically significant or substantively important effects. 

Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1 : None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Met. The one study of Saxon Elementary School Math that met WWC standards with reservations showed an indeterminate effect. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies of Saxon Elementary School Math met WWC evidence standards for a strong design. Further, no studies showed a statistically 
significant positive effect. 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies showed a statistically significant or substantively important negative effect. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : At least one study showing a statistically significant or substantively important positive effect. 

Not met. No studies showed a statistically significant or substantively important positive effect. 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Not met. No studies showed a statistically significant important effect, either negative or positive. The one study of Saxon Elementary School Math 
that met WWC evidence standards with reservations showed an indeterminate effect. 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1 : At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. 

Not met. No studies of Saxon Elementary School Math showed a statistically significant or substantively important effect, either positive or negative. 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Not met. No studies of Saxon Elementary School Math a showed statistically significant or substantively important effect. 

(continued) 
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Appendix A5 Saxon Elementary School Math rating for the mathematics achievement domain (continued) 



Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence 

• Criterion 1 : At least one study showing a statistically significant or substantively important negative effect. 

Not met. No studies of Saxon Elementary School Math showed a statistically significant or substantively important negative effect. 

• Criterion 2: No studies showing a statistically significant or substantively important positive effect, or more studies showing statistically significant or substantively 
important negative effects than showing statistically significant or substantively important positive effects. 

Met. No studies of Saxon Elementary School Math showed a statistically significant or substantively important positive effect. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1 : Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies showed a statistically significant negative effect or met WWC evidence standards for a strong design. 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. No studies of Saxon Elementary School Math showed a statistically significant or substantively important positive effect. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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